- What is reproducible research?
- Why do we care?
- Why reproducibility questions arise?
- The cost of reproducibility
- Reproducibility and statistics
- Current status of reproducibility
- What can we do?
Mikhail Dozmorov, Ph.D.
mikhail.dozmorov@vcuhealth.org
Overview
What is reproducible research?
Reproducibility and scientific progress
- Science is the systematic enterprise of gathering knowledge about the universe and organizing and condensing that knowledge into testable laws and theories
- The success and credibility of science are anchored in the willingness of scientists to expose their ideas and results to independent testing and replication by other scientists.
What is reproducible research?
- Reproducibility
- Replicability
- Repeatability
- Reliability
- Robustness
- Generalizability
What is reproducible research?
- Transparency
- Open Science
- TRUTH
What is reproducible research?
Reproducible research is the ultimate standard for strengthening scientific evidence by independent:
- Investigators
- Data
- Analytical methods
- Laboratories
- Instruments
Why do we care?
More data = more chance for errors
- High-throughput biology generates volumes of data
- Data-generating technologies are increasingly used to make clinical recommendations and treatment decisions
- A problem may be overlooked .. Published .. Get in clinical trials
More data = more chance for errors
Poor medical tests getting to patients
- OvaSure diagnostic test for ovarian cancer
Clinical trials based on flawed and fraudulent data
- Described drug response “gene signatures” in NCI60 cell lines
- Demonstrated these “signatures” correspond to patient-specific signatures and can be used to predict patient response to the drugs
Clinical trials based on flawed and fraudulent data
- Described drug response “gene signatures” in NCI60 cell lines
- Demonstrated these “signatures” correspond to patient-specific signatures and can be used to predict patient response to the drugs
Biostatisticians spot errors
“Off-by-one” error
Published Replicated
... [3,] 1881_at 1882_g_at [4,] 31321_at 31322_at [5,] 31725_s_at 31726_at [6,] 32307_r_at 32308_r_at ...
New signatures continue to be published
Initial data
-
Sample Response GSM44303 RES GSM44304 RES GSM9653 RES GSM9653 RES GSM9654 RES GSM9655 RES GSM9656 RES GSM9657 RES GSM9658 SEN GSM9658 SEN RES/SEN - resistant/sensitive
Initial data
-
Sample Response GSM44303 RES GSM44304 RES GSM9653 RES GSM9653 RES GSM9654 RES GSM9655 RES GSM9656 RES GSM9657 RES GSM9658 SEN GSM9658 SEN RES/SEN - resistant/sensitive
More data added
-
Sample Response GSM44303 RES GSM44304 RES GSM9653 RES GSM9653 RES GSM9654 RES GSM9655 RES GSM9656 RES GSM9657 RES GSM9658 SEN GSM9658 SEN RES/SEN - resistant/sensitive
-
GSM9694 RES GSM9695 RES GSM9696 RES GSM9698 RES GSM9699 SEN GSM9701 RES GSM9708 RES GSM9708 SEN GSM9709 RES GSM9711 RES
More data added
-
Sample Response GSM44303 RES GSM44304 RES GSM9653 RES GSM9653 RES GSM9654 RES GSM9655 RES GSM9656 RES GSM9657 RES GSM9658 SEN GSM9658 SEN RES/SEN - resistant/sensitive
-
GSM9694 RES GSM9695 RES GSM9696 RES GSM9698 RES GSM9699 SEN GSM9701 RES GSM9708 RES GSM9708 SEN GSM9709 RES GSM9711 RES
Summary of the Duke case
- A total of 162 co-authors
- 40 papers
- Two-thirds are partially or completely retracted
IOM guidelines on translational omics
http://www.nationalacademies.org/hmd/reports/2012/evolution-of-translational-omics.aspx
- Kaiser J “Clinical medicine. Biomarker tests need closer scrutiny, IOM concludes.” Science 2012
http://science.sciencemag.org/content/335/6076/1554.long
- McShane LM et.al. “Criteria for the use of omics-based predictors in clinical trials.” Nature 2013
http://www.nature.com/nature/journal/v502/n7471/full/nature12564.html
PubMed stats on “Reproducible research” vs. “Retraction”
“Reproducible research” “Retraction”
The cost of reproducibility
Irreproducfibility ranges from 51% to 89%
Cost of irreproducibility
Why reproducibility questions arise?
Patterns in the noise
- Humans are good at recognizing patterns
Human beings do not have very many natural defenses. We are not all that fast, and we are not all that strong. We do not have claws or fangs or body armor. We cannot spit venom. We cannot camouflage ourselves. And we cannot fly. Instead, we survive by means of our wits. Our minds are quick. We are wired to detect patterns and respond to opportunities and threats without much hesitation.
- Nate Silver
Patterns in the noise
- Humans are good at recognizing patterns
Patterns in the noise
- Humans are good at recognizing patterns
Irreproducibility in high-throughput biology
Our intuition about patterns in high dimensional data quickly drops with the increased dimensionality of the data
We rely on computation to uncover patterns
P values, the ‘gold standard’ of statistical validity, are not as reliable as many researchers assume.
The chance of being wrong
- The probability of irreproducibility of studies with p-values in the range 0.005 to 0.05 is roughly 0.33
- Regina Nuzzo, “Scientific Method: Statistical errors” Nature 2014 http://www.nature.com/news/scientific-method-statistical-errors-1.14700
- Dennis Boos and Leonard Stefanski “P-value precision and reproducibility” Am Statistician, 2001 http://www.tandfonline.com/doi/abs/10.1198/tas.2011.10129
The chance of being wrong
http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124
- Alex Tabarrok “Why Most Published Research Findings are False” 2005
http://marginalrevolution.com/marginalrevolution/2005/09/why_most_publis.html
The chance of being wrong
- In evaluating any study try to take into account the amount of background noise. That is, remember that the more hypotheses which are tested and the less selection which goes into choosing hypotheses the more likely it is that you are looking at noise.
- Bigger samples are better. (But note that even big samples won't help to solve the problems of observational studies which is a whole other problem).
- Small effects are to be distrusted.
- Multiple sources and types of evidence are desirable.
- Evaluate literatures not individual papers.
- Trust empirical papers which test other people's theories more than empirical papers which test the author's theory.
- As an editor or referee, don't reject papers that fail to reject the null.
- John Ioannidis “Why Most Published Research Findings Are False” PLOS Medicine 2005
http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124
Understanding the p-value
- Ronald Wasserstein and Nicole Lazar “The ASA's statement on p-values: context, process, and purpose” Am Stat 2016
http://amstat.tandfonline.com/doi/abs/10.1080/00031305.2016.1154108
- Monya Baker “Statisticians issue warning over misuse of P value” Nature 2016
http://www.nature.com/news/statisticians-issue-warning-over-misuse-of-p-values-1.19503
Understanding the p-value
- P-values can indicate how incompatible the data are with a specified statistical model.
- P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.
- Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.
- Proper inference requires full reporting and transparency.
- A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
- By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.
P-value warning: consult a statistician before the experiment
Current status of reproducibility
Focus on preclinical research
Focus on preclinical research
NIH focus on openness
NIH focus on openness
- Public Access Policy
- Data Sharing Policies
https://www.nlm.nih.gov/NIHbmic/nih_data_sharing_policies.html
- Genomics Data Sharing (GDS)
https://gds.nih.gov/03policy2.html
- Model Organism Sharing Policy
https://grants.nih.gov/grants/policy/model_organism/
- Research Tools Policy
NSF stance on openness
NSF stance on openness
- Responsible Conduct of Research (RCR)
https://www.nsf.gov/bfa/dias/policy/rcr.jsp
- NSF Scientific Integrity Policy
Reproducibility initiatives
Reproducibility initiatives
Reproducibility initiatives
Reproducibility guidelines
- ARRIVE – Animal Research Reporting of In Vivo Experiments
- CONSORT – Consolidated Standards of Reporting Trials
- SPIRIT – Standard Protocol Items: Recommendations for Interventional Trials
- STROBE – The Strengthening the Reporting of Observational Studies in Epidemiology
- (STARD) TRIPOD – Transparent Reporting of a multivariable prediction model for Individual PROgnosis of Diagnosis
- REMARK – REporting recommendations for tumour MARKer prognostic studies
http://www.equator-network.org/ - over 300 reporting guidelines